Skip to content

chore(infra): retire self-hosted MinIO after DO Spaces flip#4

Closed
mastermanas805 wants to merge 1 commit into
masterfrom
chore/retire-self-hosted-minio
Closed

chore(infra): retire self-hosted MinIO after DO Spaces flip#4
mastermanas805 wants to merge 1 commit into
masterfrom
chore/retire-self-hosted-minio

Conversation

@mastermanas805
Copy link
Copy Markdown
Member

@mastermanas805 mastermanas805 commented May 11, 2026

Summary

DO Spaces is now the active object-store backend (verified live: POST /storage/new returns Spaces credentials, PUT+GET round-trip works). The self-hosted MinIO Deployment in the instant-data k8s namespace is no longer in the request path. This PR retires the local MinIO manifests and replaces every MINIO_* env injection on production k8s deployments with OBJECT_STORE_* sourced from instant-secrets / instant-infra-secrets.

DO NOT MERGE UNTIL 2026-05-12 (24h after this PR opens)

Anonymous storage tokens have a 24h TTL. Any anonymous tokens minted against MinIO in the last 24h will 404 if this Deployment is removed before TTL expires. Merging before 2026-05-12 would break those still-live tokens. Authenticated (claimed) storage resources are not at risk — they were re-minted against DO Spaces at flip time. Hold this PR for at least 24h, then merge.

Files deleted

  • k8s/data/minio.yaml — Deployment, PVC (minio-data, 10Gi), cluster Service, and NodePort Service (30900 API / 30901 console)
  • k8s/data/minio-bucket-init.yaml — one-shot Job that created the instant-shared bucket via mc mb --ignore-existing
  • k8s/data/minio-secret.yaml — local-dev MinIO root creds Secret (minio-secrets in instant-data namespace)

Manifests still referencing MINIO_* (cleaned in this PR)

File Change
k8s/app.yaml (instant-api) Removed optional MINIO_ROOT_USER / MINIO_ROOT_PASSWORD secret refs. API now reads OBJECT_STORE_* from instant-secrets via internal/config/config.go.
k8s/worker/deployment.yaml (instant-worker) Replaced the storage_bytes-scanner MINIO_* block with OBJECT_STORE_* env refs sourced from instant-infra-secrets.
k8s/provisioner/deployment.yaml (instant-provisioner) Removed the MINIO_* block used for StorageBytes queries. Also fixes a dangling reference to a minio-secrets Secret in the instant-infra namespace that was never defined in-tree (the in-repo Secret lived in instant-data).
k8s/configmap.yaml Removed MINIO_ENDPOINT and MINIO_BUCKET_NAME.
k8s/secrets.yaml (template) Removed MINIO_ROOT_USER / MINIO_ROOT_PASSWORD template keys; added OBJECT_STORE_BACKEND/ENDPOINT/PUBLIC_URL/ACCESS_KEY/SECRET_KEY/BUCKET/REGION/SECURE template keys for DO Spaces.

Not touched (intentional)

  • k8s/data/minio-ingress.yaml — this file is untracked locally and not part of this PR. It serves s3.instanode.dev -> the minio Service and will 404 once the Deployment is removed. A follow-up should delete it (or repoint to DO Spaces). Flagged so it's not forgotten.
  • api/internal/config/config.go — keeps the legacy MINIO_* fallback so local docker-compose dev (where a developer might still have MINIO_* in .env) keeps working unchanged. The fallback chain is documented in-place; production no longer relies on it.

Post-merge checklist

Once this PR merges, run on the production cluster (and on any local Rancher Desktop clusters with the old MinIO still deployed):

# Confirm what's there first
kubectl get deploy,pvc,svc,job -n instant-data -l app=minio

# Remove the MinIO workload + storage + services + bootstrap Job
kubectl delete -n instant-data deploy/minio pvc/minio-data svc/minio svc/minio-external
kubectl delete -n instant-data job/minio-bucket-init --ignore-not-found
kubectl delete -n instant-data secret/minio-secrets --ignore-not-found

# If MinIO was deployed as a StatefulSet on prod (not the case in this repo's
# manifest, but worth checking), also:
# kubectl delete -n instant-data statefulset/minio

# Verify nothing left
kubectl get pods -n instant-data | grep -i minio   # should print nothing

# Sanity-check that the storage hot path is unaffected
curl -s -X POST http://<api-host>/storage/new | jq '.endpoint, .bucket'
# Expect a DO Spaces endpoint (e.g. nyc3.digitaloceanspaces.com), not a
# minio.instant-data.svc.cluster.local hostname.

Rollback plan

If DO Spaces shows any unforeseen issues after merge:

  1. Revert this PR (or git revert <merge-sha> and push to master).
  2. Re-apply the resurrected manifests: 
    kubectl apply -f k8s/data/minio-secret.yaml
kubectl apply -f k8s/data/minio.yaml
kubectl apply -f k8s/data/minio-bucket-init.yaml
  3. Roll the API and worker so they pick the MINIO_* env vars back up: 
    kubectl rollout restart -n instant deploy/instant-api
kubectl rollout restart -n instant-infra deploy/instant-worker
kubectl rollout restart -n instant-infra deploy/instant-provisioner
  4. Flip OBJECT_STORE_BACKEND=minio-admin in instant-secrets and restart instant-api to switch the credential-issuance strategy back to per-customer IAM users.

Storage data isn't lost in either direction — the PVC was on local-path storage in Rancher Desktop only; DO Spaces holds the actual production object data and was already the active backend at PR-open time.

Test plan

  • Hold this PR open until 2026-05-12 (24h after open) — anonymous storage TTLs.
  • Verify kubectl apply -f k8s/ on a fresh Rancher Desktop cluster succeeds without the MinIO manifests.
  • Verify POST /storage/new returns DO Spaces credentials (endpoint, bucket, region) after the merge.
  • Verify the worker storage_bytes scanner logs read OBJECT_STORE_* (or fail-open warn-log if scanner not yet ported).
  • Run post-merge checklist above on production cluster.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

@mastermanas805 @claude
chore(infra): retire self-hosted MinIO after DO Spaces flip
94d6be4 
DO Spaces is now the active object-store backend. The self-hosted
MinIO Deployment in the instant-data namespace is no longer in the
request path for POST /storage/new. This PR retires the local MinIO
manifests and replaces every MINIO_* env injection on production k8s
deployments with OBJECT_STORE_* sourced from instant-secrets /
instant-infra-secrets.

Manifests deleted:
  - k8s/data/minio.yaml             (Deployment + PVC + Service +
                                     NodePort 30900/30901)
  - k8s/data/minio-bucket-init.yaml (Job that created the
                                     instant-shared bucket on first
                                     deploy)
  - k8s/data/minio-secret.yaml      (local-dev MinIO root creds in
                                     instant-data ns)

Env-var injection updated:
  - k8s/app.yaml (instant-api): removed MINIO_ROOT_USER/PASSWORD
    optional secret refs. The API reads OBJECT_STORE_* from
    instant-secrets via internal/config/config.go.
  - k8s/worker/deployment.yaml (instant-worker): replaced MINIO_*
    env vars (the storage_bytes scanner block) with OBJECT_STORE_*
    sourced from instant-infra-secrets.
  - k8s/provisioner/deployment.yaml (instant-provisioner): removed
    the MINIO_* block used for StorageBytes queries. Also removes
    the dangling reference to a "minio-secrets" Secret in the
    instant-infra namespace that was never defined in-tree.
  - k8s/configmap.yaml: removed MINIO_ENDPOINT / MINIO_BUCKET_NAME.
  - k8s/secrets.yaml (template): removed MINIO_ROOT_USER/PASSWORD
    template keys; added OBJECT_STORE_* template keys.

Not touched (intentional):
  - k8s/data/minio-ingress.yaml is untracked locally and not part of
    this commit; it serves s3.instanode.dev -> minio Service and will
    404 once the Deployment is removed. A follow-up commit on master
    should delete or repoint it to DO Spaces.
  - api/internal/config/config.go keeps the legacy MINIO_* fallback
    so local docker-compose dev (where MINIO_* env vars may still be
    set in a .env file) continues to work unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@mastermanas805
Copy link
Copy Markdown
Member Author

mastermanas805 commented May 20, 2026

Rebase attempt 2026-05-20 hit conflicts. Still RELEVANT — k8s/data/minio.yaml + minio-bucket-init.yaml + minio-secret.yaml still exist on master, and configmap.yaml still has MinIO refs. Needs manual rebase against 1b65d9c (today's manifest reconciliation work likely caused the conflicts). Recommend opening a fresh follow-up PR against current master if reviewer effort here is too high.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

@mastermanas805 mastermanas805 mentioned this pull request May 20, 2026
Merged
@mastermanas805
Copy link
Copy Markdown
Member Author

mastermanas805 commented May 20, 2026

Superseded by #16 (rebased against current master after the broad manifest reconciliation landed in 1b65d9c). The new PR redoes this retirement cleanly with the post-2026-05-20 OBJECT_STORE_* abstraction wired in. Closing in favor of #16.

#16

mastermanas805 added a commit that referenced this pull request May 21, 2026
@mastermanas805 @claude
chore(infra): retire self-hosted MinIO — DO Spaces is canonical (supe…
cf16b4f 
…rsedes #4)

DO Spaces (nyc3.digitaloceanspaces.com, bucket instant-shared) has
been the active object-store backend in production since 2026-05-11.
The self-hosted MinIO Deployment in instant-data is no longer in the
request path for /storage/new — verified via live cluster:

  OBJECT_STORE_BACKEND=shared-key  (alias → do-spaces)
  OBJECT_STORE_ENDPOINT=nyc3.digitaloceanspaces.com
  OBJECT_STORE_BUCKET=instant-shared

This commit retires the local MinIO manifests and replaces every
MINIO_* env injection on production k8s deployments with OBJECT_STORE_*
sourced from instant-secrets / instant-infra-secrets.

Supersedes PR #4 (2026-05-11), which had drifted from current master
after the broad manifest reconciliation work landed.

Manifests deleted:
  - k8s/data/minio.yaml             (Deployment + PVC + ClusterIP + NodePort)
  - k8s/data/minio-bucket-init.yaml (one-shot Job creating instant-shared)
  - k8s/data/minio-secret.yaml      (local-dev MinIO root creds)

Env-var injection migrated MINIO_* → OBJECT_STORE_* in:
  - k8s/app.yaml             (instant-api) — removed MINIO_ROOT_USER/PASSWORD
                              optional refs; OBJECT_STORE_* keys already
                              wired via prior storage-abstraction work.
  - k8s/worker/deployment.yaml (instant-worker) — storage_bytes scanner
                              now reads OBJECT_STORE_* from instant-infra-secrets.
  - k8s/provisioner/deployment.yaml (instant-provisioner) — storage_bytes
                              scanner block migrated. Also removes the
                              dangling reference to a 'minio-secrets'
                              Secret in the instant-infra namespace that
                              was never defined in-tree (the in-repo
                              Secret lived in instant-data).
  - k8s/configmap.yaml — removed MINIO_ENDPOINT / MINIO_BUCKET_NAME.
  - k8s/secrets.yaml (template) — removed MINIO_ROOT_USER/PASSWORD
                              template keys; added OBJECT_STORE_*
                              template keys for DO Spaces.

Docs:
  - k8s/APPLY-CHECKLIST.md — new 'MinIO retirement' section with the
    post-merge operator cleanup commands (kubectl delete deploy/minio
    pvc/minio-data svc/{minio,minio-external} job/minio-bucket-init
    secret/minio-secrets in instant-data). Per CLAUDE.md rule 15
    (infra has no auto-apply), the operator runs this manually after
    merge.

Verification:
  - kubectl apply --dry-run=server -f k8s/{app,configmap,secrets,
    provisioner/deployment,worker/deployment}.yaml — all clean (no
    schema errors; pre-existing E2E_TEST_TOKEN hides-previous warning
    is unrelated to this change).
  - Live confirmation prod is on DO Spaces: instant-secrets has
    OBJECT_STORE_BACKEND=shared-key (alias to do-spaces),
    OBJECT_STORE_ENDPOINT=nyc3.digitaloceanspaces.com.

Coverage block:
  Symptom:       Self-hosted MinIO still deployed in instant-data while
                 prod traffic is on DO Spaces
  Enumeration:   rg -n -i 'minio' k8s/ --type yaml
  Sites found:   8 (3 manifest files + 5 env-injection blocks across
                 app/configmap/worker/provisioner/secrets)
  Sites touched: 8 (3 deleted + 5 migrated to OBJECT_STORE_*)
  Coverage test: kubectl apply --dry-run=server clean across all
                 mutated manifests (no resolveError on missing minio-
                 secrets, no env-var redefinition warnings beyond the
                 pre-existing E2E_TEST_TOKEN one)
  Live verified: Operator runs APPLY-CHECKLIST.md 'MinIO retirement'
                 section commands post-merge; prod request path
                 verified pre-PR via OBJECT_STORE_BACKEND env on live
                 instant-secrets (DO Spaces, nyc3, instant-shared).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mastermanas805 added a commit that referenced this pull request May 21, 2026
@mastermanas805 @claude
chore(infra): retire self-hosted MinIO — DO Spaces is canonical (supe…
958f7d3 
…rsedes #4) (#16)

DO Spaces (nyc3.digitaloceanspaces.com, bucket instant-shared) has
been the active object-store backend in production since 2026-05-11.
The self-hosted MinIO Deployment in instant-data is no longer in the
request path for /storage/new — verified via live cluster:

  OBJECT_STORE_BACKEND=shared-key  (alias → do-spaces)
  OBJECT_STORE_ENDPOINT=nyc3.digitaloceanspaces.com
  OBJECT_STORE_BUCKET=instant-shared

This commit retires the local MinIO manifests and replaces every
MINIO_* env injection on production k8s deployments with OBJECT_STORE_*
sourced from instant-secrets / instant-infra-secrets.

Supersedes PR #4 (2026-05-11), which had drifted from current master
after the broad manifest reconciliation work landed.

Manifests deleted:
  - k8s/data/minio.yaml             (Deployment + PVC + ClusterIP + NodePort)
  - k8s/data/minio-bucket-init.yaml (one-shot Job creating instant-shared)
  - k8s/data/minio-secret.yaml      (local-dev MinIO root creds)

Env-var injection migrated MINIO_* → OBJECT_STORE_* in:
  - k8s/app.yaml             (instant-api) — removed MINIO_ROOT_USER/PASSWORD
                              optional refs; OBJECT_STORE_* keys already
                              wired via prior storage-abstraction work.
  - k8s/worker/deployment.yaml (instant-worker) — storage_bytes scanner
                              now reads OBJECT_STORE_* from instant-infra-secrets.
  - k8s/provisioner/deployment.yaml (instant-provisioner) — storage_bytes
                              scanner block migrated. Also removes the
                              dangling reference to a 'minio-secrets'
                              Secret in the instant-infra namespace that
                              was never defined in-tree (the in-repo
                              Secret lived in instant-data).
  - k8s/configmap.yaml — removed MINIO_ENDPOINT / MINIO_BUCKET_NAME.
  - k8s/secrets.yaml (template) — removed MINIO_ROOT_USER/PASSWORD
                              template keys; added OBJECT_STORE_*
                              template keys for DO Spaces.

Docs:
  - k8s/APPLY-CHECKLIST.md — new 'MinIO retirement' section with the
    post-merge operator cleanup commands (kubectl delete deploy/minio
    pvc/minio-data svc/{minio,minio-external} job/minio-bucket-init
    secret/minio-secrets in instant-data). Per CLAUDE.md rule 15
    (infra has no auto-apply), the operator runs this manually after
    merge.

Verification:
  - kubectl apply --dry-run=server -f k8s/{app,configmap,secrets,
    provisioner/deployment,worker/deployment}.yaml — all clean (no
    schema errors; pre-existing E2E_TEST_TOKEN hides-previous warning
    is unrelated to this change).
  - Live confirmation prod is on DO Spaces: instant-secrets has
    OBJECT_STORE_BACKEND=shared-key (alias to do-spaces),
    OBJECT_STORE_ENDPOINT=nyc3.digitaloceanspaces.com.

Coverage block:
  Symptom:       Self-hosted MinIO still deployed in instant-data while
                 prod traffic is on DO Spaces
  Enumeration:   rg -n -i 'minio' k8s/ --type yaml
  Sites found:   8 (3 manifest files + 5 env-injection blocks across
                 app/configmap/worker/provisioner/secrets)
  Sites touched: 8 (3 deleted + 5 migrated to OBJECT_STORE_*)
  Coverage test: kubectl apply --dry-run=server clean across all
                 mutated manifests (no resolveError on missing minio-
                 secrets, no env-var redefinition warnings beyond the
                 pre-existing E2E_TEST_TOKEN one)
  Live verified: Operator runs APPLY-CHECKLIST.md 'MinIO retirement'
                 section commands post-merge; prod request path
                 verified pre-PR via OBJECT_STORE_BACKEND env on live
                 instant-secrets (DO Spaces, nyc3, instant-shared).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mastermanas805 added a commit that referenced this pull request May 21, 2026
@mastermanas805 @claude
chore(infra): retire self-hosted MinIO — DO Spaces is canonical (supe…
6ef8869 
…rsedes #4)

DO Spaces (nyc3.digitaloceanspaces.com, bucket instant-shared) has
been the active object-store backend in production since 2026-05-11.
The self-hosted MinIO Deployment in instant-data is no longer in the
request path for /storage/new — verified via live cluster:

  OBJECT_STORE_BACKEND=shared-key  (alias → do-spaces)
  OBJECT_STORE_ENDPOINT=nyc3.digitaloceanspaces.com
  OBJECT_STORE_BUCKET=instant-shared

This commit retires the local MinIO manifests and replaces every
MINIO_* env injection on production k8s deployments with OBJECT_STORE_*
sourced from instant-secrets / instant-infra-secrets.

Supersedes PR #4 (2026-05-11), which had drifted from current master
after the broad manifest reconciliation work landed.

Manifests deleted:
  - k8s/data/minio.yaml             (Deployment + PVC + ClusterIP + NodePort)
  - k8s/data/minio-bucket-init.yaml (one-shot Job creating instant-shared)
  - k8s/data/minio-secret.yaml      (local-dev MinIO root creds)

Env-var injection migrated MINIO_* → OBJECT_STORE_* in:
  - k8s/app.yaml             (instant-api) — removed MINIO_ROOT_USER/PASSWORD
                              optional refs; OBJECT_STORE_* keys already
                              wired via prior storage-abstraction work.
  - k8s/worker/deployment.yaml (instant-worker) — storage_bytes scanner
                              now reads OBJECT_STORE_* from instant-infra-secrets.
  - k8s/provisioner/deployment.yaml (instant-provisioner) — storage_bytes
                              scanner block migrated. Also removes the
                              dangling reference to a 'minio-secrets'
                              Secret in the instant-infra namespace that
                              was never defined in-tree (the in-repo
                              Secret lived in instant-data).
  - k8s/configmap.yaml — removed MINIO_ENDPOINT / MINIO_BUCKET_NAME.
  - k8s/secrets.yaml (template) — removed MINIO_ROOT_USER/PASSWORD
                              template keys; added OBJECT_STORE_*
                              template keys for DO Spaces.

Docs:
  - k8s/APPLY-CHECKLIST.md — new 'MinIO retirement' section with the
    post-merge operator cleanup commands (kubectl delete deploy/minio
    pvc/minio-data svc/{minio,minio-external} job/minio-bucket-init
    secret/minio-secrets in instant-data). Per CLAUDE.md rule 15
    (infra has no auto-apply), the operator runs this manually after
    merge.

Verification:
  - kubectl apply --dry-run=server -f k8s/{app,configmap,secrets,
    provisioner/deployment,worker/deployment}.yaml — all clean (no
    schema errors; pre-existing E2E_TEST_TOKEN hides-previous warning
    is unrelated to this change).
  - Live confirmation prod is on DO Spaces: instant-secrets has
    OBJECT_STORE_BACKEND=shared-key (alias to do-spaces),
    OBJECT_STORE_ENDPOINT=nyc3.digitaloceanspaces.com.

Coverage block:
  Symptom:       Self-hosted MinIO still deployed in instant-data while
                 prod traffic is on DO Spaces
  Enumeration:   rg -n -i 'minio' k8s/ --type yaml
  Sites found:   8 (3 manifest files + 5 env-injection blocks across
                 app/configmap/worker/provisioner/secrets)
  Sites touched: 8 (3 deleted + 5 migrated to OBJECT_STORE_*)
  Coverage test: kubectl apply --dry-run=server clean across all
                 mutated manifests (no resolveError on missing minio-
                 secrets, no env-var redefinition warnings beyond the
                 pre-existing E2E_TEST_TOKEN one)
  Live verified: Operator runs APPLY-CHECKLIST.md 'MinIO retirement'
                 section commands post-merge; prod request path
                 verified pre-PR via OBJECT_STORE_BACKEND env on live
                 instant-secrets (DO Spaces, nyc3, instant-shared).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mastermanas805